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ABSTRACT 


Recent  experience  at  ESD  in  acquiring  complex  computer  based  systems 
has  identified  a  deficiency  in  existing  systems  management  techniques 
in  the  area  of  computer  programs.  The  systems  management  techniques 
generally  in  use  were  designed  for  "equipment 11  systems  and  need  to  be 
expanded  to  include  computer  programs.  This  paper  describes  an  ESD 
approach  to  adapting  existing  AFSC  system  management  technioues  to 
computer  programs.  Procedures  for  insuring  system  compatibility, 
design  integrity  and  technical  control  are  discussed  and  a  method 
for  achieving  design  verification  and  qualification  is  presented. 
Particular  emphasis  is  placed  on  the  relationship  of  these  techniques 
to  computer  programs  as  elements  of  large  computer  based  systems.  The 
application  of  these  techniques  is  illustrated  through  selected  examples 
taken  from  current  ESD  system  procurements. 
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SUMMARY 


Recent  experience  at  ESD  in  acquiring  complex  computer  based  systems  nas 
identified  a  deficiency  in  existing  systems  management  techniques  in  the 
area  of  computer  programs.  The  systems  management  techniques  generally 
in  use  were  designed  for  Equipment11  systems  and  needed  to  be  expanded 
to  include  computer  programs.  This  paper  describes  an  ESD  approach  to 
adapting  existing  AFSC  systems  management  techniques  to  computer  programs. 
Procedures  for  insuring  system  compatibility,  design  integrity  and 
technical  control  are  discussed  and  a  method  for  achieving  design 
verification  and  qualification  is  presented.  Particular  emphasis  is 
placed  on  the  relationship  of  these  techniques  to  computer  programs  as 
elements  of  large  computer  based  systems.  The  application  of  these 
techniques  is  illustrated  through  selected  examples  taken  from  current 
ESD  system  procurements. 


INTRODUCTION 


Systems  Management 

The  concepts  of  "Systems  Management”  within  the  Air  Force  are  well 
established  in  the  375-series  Air  Force  regulations  and  Systems  Command 
manuals.  After  many  years  of  experience  the  procedures  for  managing 
systems  acquisition  have  become  highly  structured  and  highly  detailed 
in  some  areas.  At  the  same  time  the  systems  management  structure  has 
been  kept  general  enough  so  that  the  techniques  can  be  easily  applied  to 
any  large  system  (whether  it  be  a  missile  system,  an  electronic  system, 
an  aeronautical  system,  etc.)  and  can  be  selectively  applied  to  small 
procurements. 

Effects  of  Computer  Programs 

Since  the  development  of  the  Semi-Automatic-Ground-Environment  (SAGE) 
system  in  the  late  50* s,  the  use  of  digital  computers  and  computer  programs 
has  played  an  increasing  role  in  military  systems.  Here  at  Electronic 
Systems  Division  (ESD)  almost  every  "L"  system  developed  in  recent  years 
has  been  intimately  involved  with  computers  and  computer  programs.  In 
spite  of  the  extensive  application  of  computer  programs,  their  role  in 
systems  has  not  been  well  defined  nor  generally  understood.  Typically, 
"Systems  Management"  has  been  applied  to  the  collection  of  equipment, 
facilities,  personnel,  doc  lamentation,  etc.,  that  comprise  a  system,  without 
regard  for  the  computer  programs  within  the  system.  To  some  extent  this 
may  be  due  to  the  self-imposed  independence  of  computer  programmers  and 
analysts  from  the  engineering  discipline.  But  to  a  greater  extent  it  is 
due  to  a  lack  of  understanding  of  computer  programs  and  their  design  and 
application.  In  part,  this  lack  of  understanding  is  because  of  the 


inherent  properties  of  a  computer  program.  The  computer  program  is  an 
elusive  and  intangible  object.  It  cannot  be  readily  seen  or  felt  and 
thus  is  difficult  to  describe. 

This  lack  of  under standing  has  not,  however,  prevented  computer 
programs  from  becoming  an  important  element  of  many  current  systems. 
Unfortunately,  the  Systems  Management  techniques  have  not  yet  recognized 
the  impact  of  computer  programs  on  systems  development.  Thus  computer 
programs  in  the  past  have  failed  to  receive  the  proper  "systems11  emphasis 
required  to  effectively  utilize  them  within  a  system.  Typically,  a  system 
under  development  has  progressed  well  into  the  detailed  design  stage  and 
often  into  fabrication  while  the  computer  programmers  have  been  left  in  a 
vacuum  to  design  and  code  the  computer  program  with  only  a  minimum  of 
guidance.  Mien  the  equipment  and  computer  programs  are  integrated,  the 
problems  start:  Functions  that  each  group  (engineers  and  programmers) 
thought  the  other  was  responsible  for  often  are  not  being  performed  at 
all*  interfaces  between  computer  programs,  equipment  and  personnel  are 
incompatible •  essentially,  the  system  will  not  work.  The  result  is  often 
extensive  redesign  of  equipment  and  computer  programs  with  an  accompanying 
increase  in  cost  and  delay  in  schedule.  Frequently  all  or  most  of  the 
redesign  effort  is  placed  on  the  computer  programs  because  of  their 
inherent  flexibility.  Continued  capitalization  on  the  flexibility  of 
computer  programs  to  correct  system  deficiencies,  without  due  consider¬ 
ation  of  systems  effectiveness,  will  eventually  place  severe  limitations 
on  the  computer  programs  within  the  system. 

In  the  past  year  the  Technical  Requirements  and  Standards  Office  of 
ESD  has  undertaken  the  task  of  expanding  the  established  systems  management 
techniques  to  include  computer  programs  as  an  essential  element  of  the 
system.  The  result  of  this  effort  has  taken  the  form  of  supplements^*  * 
to  the  AFSC  375  series  manuals  that  can  be  used  as  the  basis  for  future 
revisions  to  these  manuals.  Essentially  the  activities  of  systems 
management  have  been  applied  to  the  fundamental  elements  of  systems  as 
illustrated  in  figure  1.  Typically,  an  Air  Force  System  Program  Office 
is  organized  along  the  five  sub-areas  of  management  shown  in  figure  1. 

Activities  represented  under  Program  Control  and  Procurement  and 
Production  tend  to  be  of  an  administrative  nature,  covering  such  matters 
as  budget,  schedules,  costs  and  contracting. 

As  depicted,  the  major  task  is  System  Engineering.  This  activity 
accomplishes  the  primary  job  of  maintaining  technical  control  of  the 
system  program  in  all  its  aspects. 

Configuration  Management  and  Test  and  Deployment  are  closely 
associated  with  System  Engineering,  but  are  separated  for  special 
consideration  and  responsibility.  They  pertain,  most  directly,  to  the 
major  system  elements:  equipment, facilities,  and  computer  programs. 
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The  inclusion  of  computer  programs  as  a  major  element  of  the  system, 
as  illustrated  in  figure  1,  represents  a  point  of  departure  from  the 
existing  systems  management  techniques. 


COMPUTER  PROGRAMS  WITHIN  A  SYSTEM 


What  is  a  Computer  Program 

Definitions  of  computer  programs  are  as  varied  as  the  systems  within 
which  they  function.  Within  the  context  of  systems  management ,  however , 
we  do  not  require  a  concise  technically  accurate  definition  that  is 
acceptable  to  all,  but  we  must  limit  computer  programs  to  some  class  of 
objects  that  we  can  effectively  manage.  For  this  purpose,  computer  programs 
are  defined  as  a  sequence  of  coded  instructions  and  data  contained  on 
magnetic  tape,  punched  cards  or  some  other  appropriate  medium  in  a  form 
suitable  for  insertion  into  a  digital  computer.  Within  the  contractual 
entities  defined  by  the  Air  Force,  i.e.  manufactured  products,  data 
(documentation),  and  services,  the  computer  program  possesses  properties 
similar  to  a  manufactured  product  and  similar  to  data  items.  Because  of 
the  similarities  to  equipments  and  a  desire  for  effective  technical 
control,  the  Air  Force  and  NASA  have  chosen  to  class  all  computer  programs 
as  manufactured  products,  i.e.  contract  end  items. 

A  Functional  Element.  The  computer  program  is  as  much  a  functional 
element  of  a  system  as  is  a  facility  or  a  piece  of  equipment.  The  computer 
program  usually  performs  functions  that  were  performed  by  equipment  or 
personnel  in  the  past.  It  generally  performs  these  functions  with  more 
speed  or  accuracy  than  was  previously  available  and  thus  has  become  an 
essential  element  of  the  system.  In  many  instances  the  computer  program 
is  basically  a  set  of  automatic  operating  procedures. 

System  Compatibility.  Since  the  computer  program  is  a  functional 
element  of  the  system,  it  must  interface  with  other  system  elements.  As 
with  other  system  interfaces,  all  of  the  computer  program  interfaces  must 
be  accurately  defined  throughout  the  system  design  process.  The  obvious 
interface  between  the  computer  program  and  the  computer  is  only  the 
beginning,  for  the  computer  program  will  also  interface  with  other 
computer  programs,  external  equipment  and  personnel. 

Design  Integrity.  Throughout  the  design  and  development  of  computer 
programs,  emphasis  must  be  placed  on  design  integrity.  Is  the  computer 
program,  as  designed,  cost  effective  in  terms  of  system  timing,  use  of 
available  computer  memory,  etc.?  WTill  the  computer  program  satisfy  all 
of  the  design  requirements?  These  and  other  questions  must  be  continuously 
asked  throughout  the  design  and  development  process. 


Performance  Verification .  The  complexity  of  a  computer  program 
requires  that  a  systematic  test  program  be  used  to  determine  compliance 
with  contractual  requirements.  The  performance  of  both  individual 
computer  programs  and  the  total  system  must  be  verified.  It  is  not 
unusual  for  the  testing  phase  of  computer  program  design  and  development 
to  represent  5 0 %  or  more  of  total  computer  program  costs. 

System  Implications 

The  definition  of  computer  programs  as  deliverable  contract  end  items 
is  the  basis  for  including  computer  programs  in  the  systems  approach. 
Established  systems  management  techniques  must  be  tailored,  though,  to 
effectively  manage  computer  programs  within  a  system.  The  systems 
approach  must  be  tailored  to  take  advantage  of  the  similarities  between 
equipment  and  computer  programs  while  catering  to  the  uniqueness  of 
computer  programs. 

Support  Items.  Computer  programs  require  most  of  the  support  items 
that  are  normally  required  for  equipment.  Thus  operator  handbooks, 
manuals,  etc.  must  be  written  and  verified  for  computer  programs. 

Training  requirements  must  also  be  considered  as  well  as  manning  require¬ 
ments  for  the  operational  system.  In  addition,  expendable  supplies  such 
as  magnetic  tape,  punch  cards,  etc.  must  be  made  available. 

Production.  Production,  in  the  sense  of  manufacturing  a  quantity  of 
"Chinese  copies11  of  a  piece  of  equipment,  does  not  apply  to  computer 
programs.  Once  the  initial  design  and  development  is  concluded  and  the 
computer  program  is  qualified,  production  is  completed.  Reproduction  of 
a  computer  program  on  magnetic  tape  or  a  deck  of  cards  to  obtain  an 
identical  copy  is  a  relatively  simple  and  inexpensive  process  involving 
only  peripheral  computer  equipment. 

Spare  Parts.  Unlike  equipment,  computer  programs  do  not  wear  out 
or  degenerate  as  a  function  of  time.  Unless  tampered  with,  a  sequence  of 
computer  instructions  will  continue  to  perform  the  same  function  endlessly 
until  the  computer  program  is  affected  by  some  external  source.  True, 
failure  may  occur  within  a  computer  program,  but  the  failure  is  always 
due  to  a  latent  design  deficiency.  Obviously,  then,  spare  parts  are  not 
required  and  provisioning,  useful  life,  interchangeability,  etc.  do  not 
apply.  Since  the  computer  program  does  not  require  spare  parts, 
maintenance  in  the  accepted  sense  of  the  word  does  not  apply.  There  is 
however,  a  term  Maintenance  of  computer  programs”  that  refers  to  the 
continual  process  of  correcting  latent  deficiencies  and  implementing 
modifications  within  computer  programs. 

Reliability.  Since  a  computer  program  never  wears  out  it  is  virtually 
impossible  to  predict  or  analyze  failure  rates.  Any  failure  of  the  computer 
program  is  a  latent  design  deficiency  and  its  occurrence  cannot  be  predicted. 
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It  is  obvious  then  that  a  computer  program  cannot  be  designed  for  reliability 
and  cannot  be  tested  or  evaluated  for  reliability.  Reliability  does  not 
apply  to  computer  programs  as  end  items  although  the  computer  programs 
may  be  used  to  enhance  system  reliability. 

Imp a c t .  The  definition  of  computer  programs  as  contract  end  items 
within  a  system  provides  a  vehicle  for  emphasizing  this  important  system 
element.  Sufficient  analysis  conducted  early  in  the  system  design  can 
identify  those  system  requirements  that  will  affect  the  computer  program 
design.  If,  for  example,  modular  computer  programs  are  required  or 
multiprogramming  is  needed,  these  requirements  can  be  identified  before 
the  detailed  design  of  computer  programs  commences.  This  approach  allows 
the  Air  Force  to  satisfy  many  of  the  objectives  of  DOD  directive  3200.9-^ 
such  as  establishing  firm  and  realistic  performance  specifications, 
precisely  defining  interfaces  and  responsibilities,  identification  of 
high  risk  areas,  and  establishing  firm  and  realistic  schedules  and  cost 
estimates  during  the  acquisition  of  computer  programs. 


THE  SYSTEMS  APPROACH 


Specification  of  Requirements 

In  applying  the  systems  approach  to  computer  based  systems  the 
computer  programs  must  be  given  "equal  time"  even  as  early  as  the 
conceptual  planning  for  the  system.  Early  conceptual  studies  must 
consider  the  computer  programs  as  a  vital  element  of  the  system.  The 
effects  of  performing  functions  by  automated  methods,  as  well  as  electrical 
or  manual  methods,  must  be  considered.  The  system  specification  should 
identify  all  of  the  system/design  requirements  for  the  total  system.  In 
allocating  the  system  requirements  to  system  segments  and  contract  end 
items,  extensive  analysis  of  the  trade-offs  between  equipment,  computer- 
programs  and  personnel  must  be  conducted.  The  result  of  this  system 
analysis  effort  is  to  establish  a  system  specification  that  identifies  the 
system  performance/design  requirements,  identifies  all  of  the  system  segments 
and  contract  end  items  and  allocates  the  design  requirements  to  these  contract 
end  items.  The  system  specification  represents  one  of  the  first  important 
steps  in  the  system  development  process.  As  shown  in  figure  2  the  system 
specification  is  the  basis  for  all  of  the  system  design  and  development 
effort  that  follows. 

From  the  system  specification  the  individual  contract  end  item 
specifications  are  developed.  It  is  this  preparation  of  a  design 
specification,  containing  performance/design  and  test  requirements,  that 
is  the  key  to  applying  systems  management  to  computer  programs.  The 
design-to  specification,  or  part  I  computer  program  contract  end  item 
(CPCEI)  specification,1  contains  all  of  the  performance,  design  and  test 
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DEFINITION  PHASE - >t< - ACQUISITION  PHASE 

Figure  2 


requirements  for  an  individual  computer  program  contract  end  item.  The 
specification  must  identify  and  define  all  of  the  interfaces  between  the 
computer  program  CEI  and  other  computer  program  and  equipment  CEI's.  The 
design  specif ication,  once  approved^  will  control  the  development  of  that 
computer  program.  Thus,  the  computer  program  CEI  will  be  designed  and 
qualified  against  its  individual  design  specification. 

Design  Integrity 


Throughout  the  development  process  the  customer  (in  this  case  the 
government)  should  actively  monitor  the  contractor's  efforts  in  developing 
the  system.  If  the  contractor  is  left  to  work  in  a  vacuum,  as  has 
happened  in  the  past,  the  delivered  system  often  does  not  satisfy  the  user. 
Both  the  contractor  and  the  customer  can  benefit  from  the  feedback 
provided  by  an  exchange  of  technical  information  throughout  the  design 
and  development  process.  In  recent  years  this  exchange  of  information  on 
military  systems  has  been  provided  by  the  conduct  of  technical  design 
reviews-1-  at  the  predetermined  times  in  the  process.  The  application  of 
these  design  reviews  to  computer  programs^  provides  the  technical  manager 
with  a  tool  to  assist  in  establishing  the  design  integrity  of  computer 
programs  within  a  system.  The  relationship  of  these  design  reviews  to 
the  system  development  process  is  indicated  in  figure  2. 

System  Design  Review.  The  purpose  of  this  first  review  is  to  study  the 
contractor 1 s  system  design  approach.  At  the  SDR  a  critical  examination  of 
the  system  design  is  performed  to  insure  that  a  proper  understanding  of  all 
design  requirements  exists.  An  analysis  of  contractor  documentation  in  the 
form  of  functional  diagrams,  trade-off  study  reports,  schematic  diagrams, 
initial  design  specifications,  etc.  is  conducted.  A  prime  objective  of  the 
SDR  is  to  review  the  allocation  of  functional  requirements  to  the  various 
system  segments  and  contract  end  items.  Thus,  for  computer  programs,  the 
SDR  must  insure  that  only  those  system  design  requirements  that  can  be 
realistically  satisfied  by  computer  programs  have  been  allocated  to 
computer  program  contract  end  items  (i.e.  operational,  utility,  diagnostic, 
etc.).  Prior  to  the  conduct  of  the  SDR,  trade-off  studies  concerning 
equipments  vs.  computer  programs  must  have  been  completed  to  provide  a 
cost  effective  allocation  of  design  requirements.  Satisfactory  completion 
of  the  SDR  permits  completion  of  the  Part  I  specifications  ("design  to" 
specifications)  for  all  computer  program  CEI's.  These  specifications 
form  the  basis  for  the  second  technical  review  in  the  design  process. 

Preliminary  Design  Review.  The  Preliminary  Design  Review  (PDR)  is 
usually  held  within  60  days  after  the  start  of  the  Acquisition  Phase.  The 
preliminary  design  of  the  computer  program  CEI  is  in  progress  based  on  the 
approved  "design  to"  specifications  for  the  end  item.  The  purpose  of  the 
PDR  is  to  evaluate  the  design  approach  for  the  end  item  or  group  of  end 
items  in  light  of  the  overall  system  requirements}  thus,  the  prime 
objective  of  the  PDR  is  achieving  design  integrity.  A  review  of  the 
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interfaces  affecting  the  computer  program  contract  end  item  is  an 
important  element  of  a  PDR.  Emphasis  is  placed  on  verification  of 
detailed  interfaces  with  equipment  and  with  other  computer  program 
CEI’s.  At  the  PDR  the  instruction  set  of  the  computer  to  be  used  must 
be  firmly  established.  The  programming  features  of  the  computer ,  e.g. 
interrupts,  multiprocessing,  time  sharing,  etc.  must  be  known.  All 
external  data  formats  and  timing  constraints  must  be  identified.  The 
computer  program  storage  requirements  and  data  base  design  are  reviewed 
for  technical  adequacy  at  this  time.  The  structure  of  the  computer 
program  contract  end  item  is  also  reviewed  at  the  PDR.  During  the  initial 
design  process  for  a  complex  end  item  the  requirements  of  the  Part  I 
specification  which  are  function-oriented  are  allocated  to  computer 
program  components  or  modules.  The  allocation  of  functions  to  computer 
program  components  within  the  CPCEI  is  examined  at  the  PDR.  The  primary 
product  of  the  review  at  this  level  is  establishing  the  integrity  of  the 
design  approach,  verifying  compatibility  with  the  Part  I  specification, 
and  verifying  the  functional  interfaces  with  other  contract  end  items 
in  order  that  detailed  design  of  the  computer  program  CEI  can  commence. 

Critical  Design  Review.  The  Critical  Design  Review  (CDR)  is  a  formal 
technical  review  of  the  design  of  the  computer  program  contract  end  item 
at  the  detailed  flowchart  level.  It  is  accomplished  to  establish  the 
integrity  of  the  computer  program  design  prior  to  coding  and  testing. 

This  does  not  preclude  any  coding  required  prior  to  the  CDR  to  demonstrate 
design  integrity,  such  as  testing  of  algorithms.  In  the  case  of  a  complex 
computer  program  CEI,  as  the  design  of  each  component  proceeds  to  the 
detailed  flowchart  level,  a  CDR  is  held  for  that  component.  In  this  manner, 
the  CDR  is  performed  incrementally  by  computer  program  components  and  the 
reviews  are  scheduled  to  optimize  the  efficiency  of  the  overall  CDR  for  the 
end  item  as  a  whole.  Due  to  the  varying  complexity  of  the  parallel  design 
efforts  for  computer  program  CEI  components,  it  would  be  unreasonable  to 
delay  all  of  the  components  being  developed  to  hold  one  CDR  for  the  computer 
program  end  item. 

At  the  CDR,  the  completed  sections  of  the  Part  II  computer  program  CEI 
specification  (detailed  technical  description)  are  reviewed  along  with 
supporting  analytical  data,  test  data,  etc.  The  compatibility  of  the  CPCEI 
design  with  the  requirements  of  the  Part  I  specification  is  established  at 
the  CDR.  f,Intern  interfaces  with  other  CPCEI^  and  "intra"  interfaces 
between  computer  program  components  are  examined  to  insure  compatibility. 
Design  integrity  is  established  by  review  of  analytical  and  test  data, 
in  the  form  of  logic  designs,  algorithms,  storage  allocations  and  associated 
methodology.  In  general,  the  primary  product  of  the  CDR  is  to  establish  the 
design  and  development  accomplished  as  the  basis  for  contination  of  the 
computer  program  development  cycle.  Immediately  following  the  CDR,  coding 
of  individual  components  takes  place  and  the  process  of  checkout  and 
testing  of  the  components  begins. 
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First  Article  Configuration  Inspection.  When  the  design  and  testing 
ot  the  computer  program  CEI  is  essentially  completed ,  the  Part  II 
Specification  is  available  for  review.  The  Part  II  specification 
provides  a  complete  and  detailed  technical  description  of  the  computer 
program  CEI  nas  built"  and  functions  as  the  primary  document  for  use  by 
programmers  in  correcting  errors  and  designing  changes  to  the  computer 
program  CEI.  The  technical  accuracy  and  completeness  of  the  Part  II 
specification  must  be  determined  prior  to  acceptance  of  the  document  by 
the  Air  Force.  The  First  Article  Configuration  Inspection  (FACl)  provides 
the  vehicle  for  the  required  review  of  the  Part  II  specification  and  is  an 
audit  of  the  Part  II  specification  and  the  computer  program  CEI  as  delivered. 
The  primary  product  of  the  FACI  is  the  formal  acceptance  by  the  Air  Force 
of  the  Part  II  specification  as  an  audited  and  approved  document.  Air 
Force  acceptance  of  the  computer  program  CEI  for  Category  II  testing  is 
based  on  the  successful  completion  of  the  Category  I  Test  Program  and  the 
FACI,  but  it  does  not  relieve  the  contractor  from  meeting  the  requirements 
of  the  system  specification.  Subsequent  to  FACI,  the  configuration  of  the 
computer  program  CEI  is  essentially  controlled  at  the  machine  instruction 
level  so  that  the  exact  configuration  is  available  for  Category  II  system 
testing . 

Design  Verification 

Testing,  as  defined  by  the  Air  Force,  is  divided  into^three  classes 
or  categories  of  testing,  two  of  which.  Category  I  and  II  are  important 
in  development  testing  of  Air  Force  systems  and  will  be  discussed  here. 
Category  I  tests  for  computer  program  CEIfs^are  conducted  by  the  contractor 
and  will  normally  proceed  in  such  a  way  that  testing  and  functional 
demonstrations  of  selected  functions  or  individual  computer  program 
components  can  begin  early  during  acquisition  and  progress  through 
successively  higher  levels  of  assembly  to  the  point  at  which  the  complete 
computer  program  CEI  is  subjected  to  formal  qualification  testing.  Since 
the  total  process  is  typically  lengthy  and  represents  the  major  expense 
of  computer  program  acquisition  for  the  system,  the  test  program  includes 
preliminary  qualification  tests  at  appropriate  stages  for  formal  review 
by  the  Air  Force.  While  the  tests  are  preliminary  in  nature  (they  do  not 
imply  acceptance  or  formal  qualification),  they  do  serve  the  necessary 
purposes  of  providing  check  points  for  monitoring  the  contractor^ 
progress  tovTards  meeting  design  objectives  and  of  verifying  detailed 
performance  characteristics  which,  because  of  sheer  numbers  and  complexity, 
may  not  be  feasible  to  verify  in  their  entirety  during  formal  qualification 
testing.  Category  II  tests  are  complete  system  tests,  including  the 
qualified  computer  program  end  items,  conducted  by  the  Air  Force  with 
contractor  support  in  as  near  an  operational  configuration  as  is  practicable. 
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Category  I  (Qualification)  Testing*  The  Category  I  test  program 
verifies  that  the  computer  program  contract  end  item  satisfies  the 
design/performance  requirements  of  the  Part  I  11 design  to1’  specification. 

The  Category  I  test  program  must  be  designed  to  insure  that  all  of  the 
functional  requirements,  as  translated  into  computer  program  components, 
are  tested  and  that  requirements  are  not  lost  in  the  translation.  The 
program  is  divided  into  two  major  classes  of  tests:  Preliminary 
Qualification  Tests  (PQT)  and  Formal  Qualification  Tests. 

Preliminary  Qualification  Testing  (PQT) .  Preliminary  qualification 
tests  are  designed  toverify  the  performance  of  individual  components  prior 
to  an  integrated  formal  qualification  of  the  complete  computer  program  CEI. 
The  PQT  phase  is  conducted  incrementally  by  components  in  the  same  manner 
as  the  Critical  Design  Review.  Figure  3  depicts  the  relationship  between 
CDR  and  the  Category  I  test  program.  The  crosshatched  blocks  in  Figure  3 
indicate  coding  of  individual  computer  program  components.  The 
Preliminary  Qualification  Tests  are  modular  and  a  “building  block11 
effect  occurs  as  testing  progresses.  As  each  computer  program  component 
is  added  and  each  PQT  conducted,  increased  confidence  develops  in  the 
computer  program  CEI  being  tested. 

Formal  Qualification  Testing  (FQT) .  Formal  qualification  tests 
represent  the  iinal  step  in  Category  Y  testing  of  a  computer  program  CEI. 
They  insure  that  the  complete  CEI  actually  meets  the  Part  I  specification 
performance  requirements.  However,  the  necessity  for  formal  qualification 
places  more  stringent  requirements  on  the  computer  program ’s" environment" ; 
it  must  now  leave  the  confines  of  an  artificial  world  and  enter  the  realm 
of  the  real  world. 

Qualification  testing  of  a  complex  computer  program  contract  end 
item  requires  extensive  use  of  simulation  techniques.  The  use  of  these 
techniques  is  dictated  by  the  high  cost  of  providing  overhead  computer 
facilities  or  by  the  unavailability  of  new  computers  undergoing  a  parallel 
design  and  development  effort.  Although  Preliminary  Qualification  Tests 
will  make  maximum  use  of  simulation  techniques,  the  Formal  Qualification 
Tests  will  generally  require  live  inputs,  live  outputs  and  operationally- 
configured  equipment.  A  prerequisite,  then,  of  FQT  is  usually  the 
installation  and  checkout  of  the  computer  program  CEI  in  an  operationally- 
configured  system  at  the  Category  II  test  site.  To  provide  reliable 
data  during  FQT  of  a  computer  program  CEI,  fully  installed  and  checked  out 
equipment  should  be  available.  Subsequent  to  installation  and  checkout 
of  the  computer  program  CEI,  FQT  is  conducted.  The  conclusion  of  FQT 
signals  the  end  of  the  Category  I  test  program.  The  computer  program  CEI 
should  be  fully  qualified  and  all  of  the  requirements  of  the  Part  I 
specification  should  be  satisfied  except  for  those  requirements  of  the 
Part  I  specification  that  can  only  be  demonstrated  during  a  Category  II 
system  test.  After  successfully  passing  this  phase  of  testing,  the  computer 
program  is  fully  integrated  into  the  system  and  is  ready  for  system  testing. 
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Figure  3 


Category  II  (System)  Testing,  At  the  conclusion  of  Category  I 
testing,  the  Air  Force  conducts  an  extensive  Category  II  system  test 
program  with  the  objective  of  demonstrating  that  the  system  meets 
system  performance/design  requirements  of  the  System  Specification. 
Insofar  as  the  computer  programs  are  concerned,  Category  II  testing 
will  verify  their  compatibility  with  the  system  and  their  integrated 
performance  in  meeting  system  requirements  in  the  live  environment, 
with  operational  communications,  personnel,  etc.  Residual  and  design 
errors  discovered  in  this  phase  of  testing  are  corrected  by  the 
contractor  prior  to  the  system  becoming  operational. 


SOME  TYPICAL  EXAMPLES 


An  Expensive  Education 

In  the  past  the  Air  Force  has  had  some  painful  experience  in  the 
acquisition  of  computer  based  systems.  In  the  acquisition  of  one  recent 
large  computer  based  system,  problems  generated  in  the  development  of  the 
information  processing  segment  proved  so  great  that  they  prevented  the 
system  from  becoming  operational.  One  of  the  factors  that  contributed  to 
the  failure  of  the  project  was  the  lack  of  a  system  specification.  As  a 
result  no  explicitly  defined  nor  commonly  understood  set  of  system 
objectives  was  established.  Consequently,  specific  system  requirements 
could  not  be  allocated  to  individual  computer  programs.  The  design  docu¬ 
ments  for  the  computer  programs  were  generally  not  definitive  enough  and 
not  subject  to  controls.  The  net  effect  was  that  a  design  requirements 
baseline  was  never  established.  The  contractor  was,  to  a  certain  degree, 
left  to  design  and  develop  the  computer  programs  on  his  own  without 
detailed  guidance  and  feedback  from  the  Air  Force  resulting  in  computer 
programs  that  never  did  satisfy  the  users  requirements  even  after  repeated 
redesigns.  The  cost  of  the  computer  programs  grew  to  five  times  the 
original  estimates  during  the  design  and  development.  When  the  decision 
was  made  to  delete  the  system  from  the  inventory  approximately  $27,000,000 
had  been  spent  on  the  system,  of  which  about  55%  represented  computer 
program  costs,  and  additional  funds  were  needed  to  meet  the  system 
requirements.  Even  those  elements  of  the  system  that  operated  properly 
suffered  from  inadequate  and  inaccurate  documentation  and  thus  could  not 
be  used  to  maximum  effectiveness.  It  cannot  be  claimed  that  the  use  of 
a  specific  technique  or  group  of  techniques  would  have  solved  all  of  the 
problems  associated  with  this  system.  It  is  clear,  however,  that  the  use 
of  a  systems  approach  that  included  the  computer  programs  would  have 
greatly  increased  the  probability  of  achieving  the  design  goals. 

Some  Recent  Improvements 

BUIC,  an  acronym  for  Back-Up  Interceptor  Control  System,  is  an  air 
defense  system  that  enhances  the  survivability  of  North  America* s  air 
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defense  in  a  post  attack  environment.  The  BUIC  system  has  progressed  in 
evolutionary  steps  from  the  manual  BUIC  I  system  to  the  sophisticated 
BUIC  III  system  that  incorporates  a  large  modular  digital  computer.  The 
BUIC  system  represents  the  first  attempt  at  ESD  to  apply  systems  management 
to  a  total  computer  based  system. 

Draft  versions  of  the  ESD  Exhibit‘d  on  configuration  management  of 
computer  programs  were  applied  to  the  BUIC  contracts.  The  system 
specification  identified  four  computer  program  contract  end  items: 

Air  Defense  Computer  Program  CEI,  Utility  Computer  Program  CEI,  System 
Exercise  Computer  Program  CEI,  and  Confidence-Diagnostic  Computer  Program 
CEI.  Specific  system  requirements  were  allocated  to  each  of  the  computer 
program  CEI*s  and  detailed  design  requirements  with  qualitative  measures, 
where  possible,  were  established  for  each  CEI.  Fault  detection  and 
isolation  are  typical  examples  of  the  requirements  placed  on  the 
Confidence/Diagnostic  CEI  while  requirements  such  as  preparation  of 
exercise  tapes  and  control  of  system  exercise  missions  were  placed  on  the 
System  Exercise  CEI.  In  addition  the  interfaces  between  the  various 
computer  program  and  equipment  CEI’s  were  defined  in  the  individual 
design  specifications.  Particular  emphasis  was  placed  on  the  interface 
between  the  Air  Defense  and  Confidence/Diagnostic  computer  programs  for 
two  reasons:  first,  the  unique  design  of  BUIC  wherein  two  programs 
operate  in  parallel  (Air  Defense  and  Confidence/Diagnostic)  using  the 
redundant  computer  modules  creates  an  exceptionally  complex  interface j 
second,  the  Confidence/Diagnostic  program  was  written  by  the  computer 
manufacturer  while  the  rest  of  the  computer  programs  wrere  written  by  a 
second  contractor. 

A  Modular  Computing  System.  It  was  during  a  BUIC  II  design  review 
that  the  full  system  impact  of  computer  programs  was  discovered.  The 
equipment  manufacturer  had  concluded  that  the  system  reliability 
requirements  could  not  be  met  with  the  proposed  equipments.  Even  the  use 
of  system  redundancy  would  not  provide  the  desired  system  Mean  Time 
betv/een  failure  (MTBF)  with  the  available  equipment  end  items.  Analysis 
of  the  modular  equipment  and  the  Confidence/Diagnostic  computer  programs 
led  to  a  computer  program-controlled  modular  computing  system  with 
redundant  modules.  The  system,  described  by  Blanton,  is  illustrated  in 
figure  [4  and  shows  the  redundant  modules.  During  normal  operation  the 
control  of  the  computing  modules  is  shared  by  two  computer  programs 
operating  in  parallel:  an  operational  (Air  Defense)  program  and  a  back-up 
(Confidence/Diagnostic)  program.  The  operational  program  performs  air 
defense  functions,  limited  failure  detection  and  system  reconfiguration 
and  startover.  The  back-up  program  exercises  the  modules  in  the  back-up 
system  and  detects  failures  to  maintain  confidence  in  the  redundant 
modules  should  they  be  needed  in  the  "operational"  system.  To  guard 
against  failures  in  the  "operational"  system  remaining  undetected  for 
long  periods  of  time  the  modules  are  periodically  switched  between  the 
"operational"  and  "back-up"  systems.  In  BUIC  the  complete  failure 
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BUIC  CENTRAL  COMPUTING  MODULES 


Figure  4 
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detection,  failure  isolation  and  system  recovery  is  accomplished  in 
less  than  30  seconds.  The  30  second  recovery  time  does  not  seriously 
degrade  system  operation  since  the  "operational"  system  is  capable  of 
restarting  where  it  was  halted  without  much  loss  of  information. 
Blanton^  has  calculated  the  MTBF  of  the  modular  system  in  figure  h  to 
be  87*858  hours.  This  represents  a  system  33  times  more  reliable  than 
a  duplexed  system,  yet  the  duplexed  system  requires  a  23%  increase  in 
modules . 

In  this  example  the  full  value  of  computer  programs  within  a  system 
was  realized  early  in  the  design  stages,  but  only  as  a  result  of  the 
equipment  being  unable  to  meet  design  requirements. 

Conclusion 


The  broadening  of  systems  management  to  include  computer  programs  as 
well  as  equipment,  facilities,  etc.  can  provide  a  powerful  tool  for  the 
technical  manager.  These  techniques  can  provide  definitive  design 
requirements  and  increase  technical  control  in  the  design  and  development 
of  computer  programs.  They  improve  the  computer  programmer's  position  in 
the  design  process  by  insuring  that  his  requirements,  interfaces,  etc.  are 
properly  emphasized  from  the  system's  conception  through  its  attainment 
of  operational  status. 

In  applying  the  systems  approach  one  cannot  equate  computer  programs 
with  equipments;  rather,  the  requirement  for  similar  technical  controls 
must  be  recognized  with  due  consideration  for  the  inherent  differences 
between  computer  programs  and  eouipnent.  To  realize  the  full  value  of 
computer  programs,  increased  emphasis  must  be  placed  on  systems  analysis 
of  computer  programs.  As  shown  in  the  BUI.C  system.,  the  potential  of 
computer  programs  is  often  far  above  that  initially  considered  and 
achieving  this  potential  may  recuire  much  innovation.  It  is  only 
through  a  systems  approach  that  the  full  systems  effectiveness  of 
computer  programs  will  be  realized. 
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